Many businesses have realized that they need to test their initiatives before rolling out the initiatives to their entire business networks. In conducting those tests, they often rely on test versus control analysis, where the business initiative is implemented at a small group of locations within their business network, and then the performance of those locations is analyzed against the performance of a set of “control” locations intended to match the test locations as closely as possible.
To get a clear understanding of the impact of the changes on the test sites, it becomes critical to compare the test sites with control sites where the same changes have not been made. These control sites should be as similar as possible to the test sites in order to lower the measurement error.
The business initiatives may intend to increase profits by a small margin—sometimes less than 2% or 3%. For a retail establishment, an increase in profits by just a few percentage points can have important impacts on the overall performance of the business. Indeed, a 1% increase in profits for a retailer with $2 billion in profit would yield $20 million, a result that could pay for the costs of the initiative many times over.
As a result, it is important to have an accurate reading of the test of the initiative and to ensure that strange or extraneous occurrences that impact a test or control location's sales or profits during the test period do not unduly influence the overall analysis of the test.
Conventionally, if there is a test or control location that experiences a extraneous occurrence, such as an extended closure due to a hurricane, the impact on the data from that location may result in all of the performance data for that test period being considered an “outlier,” and the entire location will be excluded from the analysis. Excluding the entire location from the analysis may result in losing valuable data points associated with time periods where the extraneous occurrence did not have an impact on that excluded location and where the location did not exhibit outlier behavior. A user may not be able to accurately identify an outlier, or the user may not be able to identify every outlier.
Another problem can occur when a test or control location is subject to an extraneous occurrence that may affect one or more days within a test period, such as police activity near the location that curtails customer traffic to the location over a day or more, but the overall effect of that occurrence is not significant enough to make the overall data associated with the location during the test period appears as an outlier, even if the extraneous occurrence had a true impact on the location's performance. In this situation, an analysis of the test period might be inaccurate. If the extraneous event depressed sales in a test location, the results of the test might not show the true increase (or decrease) in sales attributable to the test.
Conventional solutions employ a computer system that uses a filter that excludes data points beyond a predetermined threshold. This automatic exclusion of data based upon a filter results in at least two problems. First, the data may actually be significant data that should be included in the analysis. For example, a promotion at a store may have a lot of success, which appears as a spike in the data, but should be included in the analysis. Second, a threshold for one location may not be an appropriate threshold for another location. For example, a store in an urban environment may have a higher magnitude of sales than a store in a rural environment, so a threshold for the urban store should be higher than a threshold of the rural store.
In order to address the deficiencies with these conventional solutions, a user must specially program the computer system to accommodate for different scenarios. This special programming relies upon the user's intuition to identify potential issues (e.g., reasons for sharply increased or decreased performance). Aside from being inefficient due to the high volume of data, an algorithm based upon a user's intuition is not accurate enough.